The process of developing technology often involves two parallel efforts: (1) the development of a new technology and (2) the development of peripheral tools for the performance analysis and behavior evaluation of the technology itself. While the field of computer science provides uncountable examples of technologies that have been developed hand in hand with such peripheral tools, two of these technologies have taken a pivotal place during the past fifty years: the computer and the Internet.
There exist at least two broad reasons for the need of analytical tools to measure the performance and behavior of Internet computing systems. First, at the early stages of development, systems are simple and their performance can usually be characterized through direct naked-eye observation. As the technology matures, its complexity increases, often to a point where its behavior is no longer easy to predict. The Internet provides a good example of such evolutionary transition. Its current behavior is both a function of (arguably) predictable computer behavior and unpredictable psychology-based human behavior. Such is the case that computer scientists have long tried to model its behavior borrowing tools from branches of math such as stochastic processes, game theory or even fractal and chaos theory. (A. B. Mackenzie, S. B. Wicker, Game theory in communications: motivation, explanation, and application to power control, IEEE Global Telecommunications Conference, 2001; M. E. Crovella, A. Bestavros, Self-similarity in World Wide Web Traffic: Evidence and Possible Causes, IEEE/ACM Transactions on Networking, 1996.) This evolutionary need for performance analysis tools arises in most of the technologies that achieve certain complexity.
Second, for the past fifty years, our economic and social superstructures have evolved to a point where almost any transaction (both economic and social) requires some form of involvement of the Internet and our computer systems. Key resources such as energy, water, communication or the stock market, to name a few, depend on the proper functioning of these two technologies. Such is the case that they are recognized as national security infrastructures subject to possible cyber attacks. (V. Paxson, Bro: A System for Detecting Network Intruders in Real-Time, Proceedings of the 7th USENIX Security Symposium, 1998.) To protect the well-being of our society, it is therefore crucial to dispose high-performance peripheral tools capable of analyzing the behavior of the networks to detect malicious usages.
Current architectures of these analysis tools are being driven to a breaking point by two independent challenges: first, as network data-rates increase, these tools are being overwhelmed by the quantity of computation they must perform to continuously analyze the network; second, as computer network systems become more sophisticated, the parsing of the network flows requires ever more complex traffic analysis heuristics that further stress the system's processing capacity.
Therefore there exists a need for a set of systems and methods that focus on the high-performance implementation of peripheral tools.